The utility of website-based quality improvement tools for health professionals: a systematic review

Abstract As technology continues to advance, it is important to understand how website-based tools can support quality improvement. Website-based tools refer to resources such as toolkits that users can access and use autonomously through a dedicated website. This review examined how website-based tools can support healthcare professionals with quality improvement, including the optimal processes used to develop tools and the elements of an effective tool. A systematic search of seven databases was conducted to include articles published between January 2012 and January 2024. Articles were included if they were peer reviewed, written in English, based in health settings, and reported the development or evaluation of a quality improvement website-based tool for professionals. A narrative synthesis was conducted using NVivo. Risk of bias was assessed using the Mixed Methods Appraisal Tool. All papers were independently screened and coded by two authors using a six-phase conceptual framework by Braun and Clarke. Eighteen studies met the inclusion criteria. Themes identified were tool development processes, quality improvement mechanisms and barriers and facilitators to tool usage. Digitalizing existing quality improvement processes (n = 7), identifying gaps in practice (n = 6), and contributing to professional development (n = 3) were common quality improvement aims. Tools were associated with the reported enhancement of accuracy and efficiency in clinical tasks, improvement in adherence to guidelines, facilitation of reflective practice, and provision of tailored feedback for continuous quality improvement. Common features were educational resources (n = 7) and assisting the user to assess current practices against standards/recommendations (n = 6), which supported professionals in achieving better clinical outcomes, increased professional satisfaction and streamlined workflow in various settings. Studies reported facilitators to tool usage including relevance to practice, accessibility, and facilitating multidisciplinary action, making these tools practical and time-efficient for healthcare. However, barriers such as being time consuming, irrelevant to practice, difficult to use, and lack of organizational engagement were reported. Almost all tools were co-developed with stakeholders. The co-design approaches varied, reflecting different levels of stakeholder engagement and adoption of co-design methodologies. It is noted that the quality of included studies was low. These findings offer valuable insights for future development of quality improvement website-based tools in healthcare. Recommendations include ensuring tools are co-developed with healthcare professionals, focusing on practical usability and addressing common barriers to enhance engagement and effectiveness in improving healthcare quality. Randomized controlled trials are warranted to provide objective evidence of tool efficacy.


INTRODUCTION
Quality improvement (QI) involves stakeholders devising plans to enhance current practices to improve outcomes [1].It is an ongoing process to continually improve practices and is often required as part of accreditation [2].A range of frameworks are available to guide the approach to QI.For example, the 'Plan-Do-Study-Act' framework allows health departments to systematically compare current practices against established standards and to modify practices for continuous QI [2].In addition, there are QI tools available.QI tools are specifically referring to instruments, such as checklists and process maps, used to implement and measure improvement initiatives, whereas QI frameworks provide systematic approaches for guiding the overall process of QI [3].
Technological advances have allowed tools to be integrated with online platforms.Research exploring how technology can support QI in a hospital setting concluded that such technology allowed for the provision of efficient and adequate feedback of performance [4].For example, electronic health records integrated with real-time data analytics enable clinicians to receive immediate feedback on patient outcomes, facilitating prompt adjustments to care protocols and improving overall patient safety [5].In addition, websites provide several advantages over other types of online platforms, such as more flexibility, accessibility and cross-platform compatibility [6].
As technology continues to advance and digitalization becomes more common, it is important to understand how digital tools can support QI in healthcare.There is abundant literature on QI frameworks [7][8][9]; however, to our knowledge, there is no literature summarizing QI tool design processes or components that can be used to guide tool development.With the advancement of technology, many digital tools will continue to be developed.QI projects can be complex and websites are a good technological solution to assist in these processes.
This systematic review aims to understand how websitebased tools can support health professionals with QI.The primary research question is: What are the optimal websitebased QI tool design processes and elements of an effective QI tool?

Protocol and registration
This review adheres to the Preferred Reporting Items for Systematic review and meta-Analysis (PRISMA) statement [10] and prospectively registered with PROSPERO no.CRD42023451346.The completed PRISMA checklist is provided in supplementary file 1.
Institutional ethics board approval was not required for this study since it was a systematic review and did not involve human subjects as research participants.

Eligibility criteria
Papers were included if they were peer reviewed, written in English, available in full text, based in a health setting and reported the development or evaluation of a website-based QI tool used by health professionals.
In this study, website-based tools were defined as resources such as toolkits that users can access and use autonomously through a dedicated website.Studies that examined tools on other platforms, e.g.mobile applications (tools specifically designed as a mobile application, as these are different to websites that are also mobile-friendly) were excluded.This exclusion was due to the distinct nature of mobile applications which typically depend on the hardware and operating systems of mobile devices and require more platform-specific programming, potentially affecting the consistency and generalizability of results [11].

Data sources and search strategy
A computerized search was conducted in December 2022 and repeated in January 2024 using PubMed, MEDLINE, ScienceDirect, Wiley, Scopus, ProQuest, Education Resource Complete, and A+ Education.Published research in health information and technology emerged in 2008 but began increasing from 2012 [12].Therefore, the databases were searched from January 2012 to January 2024.Search terms used were quality improvement tool OR quality tool AND web-based OR website-based.The complete search strategy is outlined in Supplementary File 2. Reference lists were also screened.
Identified articles were uploaded and duplicates removed in Covidence [13].Titles and abstracts were reviewed by one author (G.T.).All potentially relevant full-text articles were independently assessed by two authors (among G.T., M.H., and J.N.).Any differences were discussed and resolved between reviewers.

Data extraction
Data extracted by one author (G.T.) via Covidence included title, author, location, date, discipline area, study design, and study population.Two authors (G.T. and M.H.) independently extracted data on tool development process, QI mechanism, tool features/resources, and outcome measures and assessments.Differences were discussed and resolved between reviewers.
Tool development data specifically examined four key steps-research/literature review, use of theoretical framework to guide development, co-developing with stakeholders, and field testing.These steps were based on the UK Medical Research Council framework for developing and evaluating complex interventions [14].

Data synthesis
All data were entered into Covidence and narratively synthesized on tool development processes, elements of an effective QI tool, and outcome measures and assessments.Studies could contribute information on one or more of these aspects.Thematic analysis using NVivo [15] followed a six-phase process [16].All papers were independently coded by two authors and cross-checked for any discrepancies.A meta-analysis was not possible as there was a high degree of heterogeneity.

Quality assessment
The Mixed Methods Appraisal Tool version 2018 [17] was used.The risk of bias was assessed independently by two authors (G.T. and M.H.).Any differences were discussed and resolved between the authors.The GRADE framework [18] was used to assess the overall body of evidence.

RESULTS
A total of 5308 articles were screened and 18 studies met the inclusion criteria (Fig. 1).
Table 1 summarizes the study characteristics.Studies that only described tool development included qualitative studies (n = 3) and a quantitative descriptive study (n = 1).Studies that reported tool evaluation included qualitative studies (n = 6), quantitative descriptive studies (n = 2), feasibility studies (n = 5), and a non-randomized experimental study (n = 1).No studies were randomized controlled trials.
Four studies only described the tool development; therefore, outcome measures were not reported [19][20][21][22].The outcomes of the remaining 14 studies are summarized in Table 2.All 14 studies reported positive outcomes.Outcome measures varied across studies as they were dependent on the tool's purpose.The various purposes of tools included improving clinical practice and compliance (n = 5), enhancing usability of traditional, paper-based or manual methods (n = 4), and supporting quality assurance and reflective practice (n = 5).

Study quality
Risk of bias results are outlined in Supplementary File 3. Most studies (n = 16) had clear research objectives and appropriate methods.Four studies described tool development only and lacked formal data analyses.While most qualitative studies justified their approach, three studies lacked detail regarding the methodology.Assessing some quantitative studies were difficult due to lack of detailed methodology, sampling strategy, or measurements.

Narrative synthesis
There were three main themes that emerged: tool development processes, QI mechanisms, and barriers and facilitators to tool usage.Overall synthesis quality, as per the GRADE framework [18], was low due to the incorporation of qualitative studies.

Tool development processes
Eleven studies described the tool development process.Four studies undertook research or literature review prior to developing the tool, one study utilzsed a theoretical framework to guide tool development, nine studies co-developed the tool or consulted with stakeholders in the development process and four studies conducted field testing.
The most adopted process was co-developing with stakeholders.The approaches varied across studies, reflecting different levels of stakeholder engagement and adoption of co-design methodologies.For instance, two studies formed specialized working groups, emphasizing collaboration among healthcare practitioners with expertise in the relevant fields [19,21].Another study involved cyclical feedback from various stakeholders, including healthcare associations and quality improvement bodies, reflecting a broader engagement strategy [23].Two studies demonstrated collaboration between professional societies and advisory groups to develop tools tailored to specific clinical domains [22,24].The least commonly adopted process was utilizing a theoretical framework (n = 1).The development of the 'Cancer Care Quality Improvement Toolkit' [25] was guided by Roger's Diffusion of Innovation Model [26], and analyses

Study population (type of professional)
Outcome measures

Key results
Crossland 2014, Australia [23] GP staff Readability, content validity and staff perceptions/process validity of the tool Readability was assessed using the Flesch-Kincaid.Readability Formula and Gunning-Fog Index in a combined online test.
Readability, content validity and process validity were assessed using a series of Likert scale questions.Some definitions were regarded as complex and many staff (except practice managers) found element descriptions hard to understand.

GP perspectives (qualitative).
There was a high level of compliance with the Methadone Treatment Protocol guidelines by the participating GPs.The survey was well received and it was stated that it encouraged reflective practice.Some suggestions for improvement were provided.
It is recognized that there are many complexities in providing treatment and care and that it would be useful to explore this through a socio-cultural lens.It is proposed that this audit process could be modified for the treatment of other chronic diseases in GP.

Clinical ethics consultants
Experience with the online comment system approach to clinical ethics consultation Analysis of consultation logs (case summaries, recommendations, and comments) to assess how system was used and the extent that it achieved goals of consensus, quality assurance and education.The online ethics comment system facilitated broad committee participation and consensus building.
The electronic medium allows for meaningful discussion and deliberation of cases and recommendations.The process assists in improving the quality of ethics consultations, ensuring that relevant details are considered and that the recommendations made are based on input from committee members.
(continued) System Usability Score of 68.57(above average).Functional and user interpretation issues were identified, such as unnecessary information, lack of system clarity, and redundant data fields.These insights were gained from the interview protocol, not the System Usability Score.Design elements like drop-down menus, free-text entry, checkboxes, and prefilled or auto-populated data fields were perceived as useful for enhancing system navigation and facilitating adverse drug reaction reporting.

Assess usability
Quantitative survey, System Usability Scale (SUS) and Net Promotor Score (NPS).The results reflected overall ease of use with need for only minor improvements (SUS score of 73.2; minimum 47.5, maximum 92.5).A SUS score >68 is considered above average in overall measure of system satisfaction and sub-scales of usability and learnability.Respondents reported that they are likely to refer colleagues to the Dashboard (NPS score 20.7).
All but 1 respondent learned something new from the dashboard, and 77% felt that the dashboard enhanced their understanding of patients' emergency department experiences.
were guided by the Consolidated Framework for Implementation Research [27].

QI mechanisms
Table 3 summarizes the primary QI process/mechanisms that the tools utilized.These included digitalizing current QI processes (transforming traditional, paper-based, or manual methods into digital formats) (n = 7), identifying gaps in practice (n = 6), professional development (n = 3), and using clinical governance and organizational management as part of QI (n = 2).Some studies described the benefits of these processes, for example, digitalization was found to be feasible and preferred over paper-based methods [28,29].These processes supported healthcare professionals by enhancing accuracy and efficiency in clinical tasks, improving adherence to guidelines, facilitating reflective practice, and providing tailored feedback for continuous quality improvement.For example, the EQUSUM tool [28] for endometriosis surgery significantly improved classification accuracy and usability compared to paper methods.There were a variety of QI features.The most common were education/training resources (n = 7), assisting the user Tool (CATT) [52] to assess current practices against standards or recommendations (n = 6), and recording activity or performance (n = 6).For example, the 'Safety Profile Assessment' [24] helped assess compliance with safety/quality indicators in radiation therapy using a 5-point Likert scale.Other features reported were an automated calculation/scoring system (n = 4), such as the 'Paediatric Endoscopy Global Rating Scale' [22] where each measure was assigned a level from D to A and a score was generated for each standard.In addition, action plans or recommendations for QI (n = 3), downloadable forms (n = 4), self-assessment of current practices (n = 2), and platforms for sharing data or information (n = 4) were reported.Some studies elaborated on how the tool's features were advantageous in QI and resulted in better clinical outcomes, increased professional satisfaction, and streamlined workflow.The 'Web-Based Tool to Report Adverse Drug Reactions' [30] allowed for recording of adverse events with all the necessary information for submission to the Australian Therapeutic Goods Administration, in line with professional requirements.The 'self-audit of methadone treatment' [31] tool could analyse the inputted data and provide a comparison to the expected standards.

Barriers and facilitators
Thirteen studies reported barriers and facilitators to tool usage (Table 4).There were three main factors contributing to the facilitators in tool usage.
Relevance to practice: the tool was considered useful and aligned with professional standards (n = 5).For example, General Practice staffs who used the 'self-audit of methadone treatment' [31] found that the tool was helpful in reflection of their clinical practice and served as a reminder of the importance of standards.The staff that used the 'Primary Care Practice Improvement Tool (PC-PIT)' [23] thought that the tool was relevant to everyday practice work and planning, particularly as the tool addressed clinical governance.
Accessibility: the tool was user-friendly and enhanced current practices through a web-based interactive platform.For example, endometriosis experts who utilized the EQUSUM tool [28] found that the tool was easy to use and there were visual advantages of using anatomical pictures for classification.
Facilitating multidisciplinary action: the tool allowed for shared decision-making and increased engagement with stakeholders.For example, one of the identified facilitators of the 'PC-PIT' tool [23] was that it allowed all staff to be involved in the identification of areas for improvement.The 'Paediatric Endoscopy Global Rating Scale' [22] found that the tool increased engagement with surgical colleagues on delivery of endoscopy services.
There were four main factors contributing to the barriers in tool usage.
Time consuming: the tool took users too long to use.For example, some users found the 'Residency Performance Index' [20] tool time-consuming to use and the 'self-audit of methadone treatment' [31] tool lengthy and protracted.
Irrelevant to practice: the tool did not provide information specific to practice, or there were concerns with implementation in the real-life setting.The 'PC-PIT' tool [23] covered areas that some users thought may be outside clinical management processes.The 'PROP' tool [32] study identified a technical barrier where the tool was unable to be integrated into electronic health records.
Difficult to use: the information offered in the tool was difficult to understand, or technical difficulties were experienced with using the tool.The 'PC-PIT' tool [23] contained elements that were difficult to understand for some users.The 'self-audit of methadone treatment' [31] had poor clarity of questions for some, and 'Quality Innovation Networks' [33] was not considered easy to navigate or use by some.
Lack of organization engagement: there were organization barriers that prevented implementation of the tool.The 'Cancer Care Quality Improvement Toolkit' [25] study identified that there was some reluctance to change and the tool required approvals for implementation that prevented wider use.The 'Safety Profile Assessment' tool [24] study found that resource constraints and clinical pressures prevented tool usage.

Statement of principal findings
The findings highlight that identifying gaps in practice, digitalizing existing processes and contributing to professional development were key mechanisms that tools adopted.These tools were associated with reported enhancement of efficiency in clinical tasks, improvement in adherence to guidelines, facilitation of reflective practice, and provision of tailored feedback for continuous quality improvement.The most common tool features were education/training resources and the ability to assist the user in assessing current practices, which supported healthcare professionals in achieving better clinical outcomes, increased professional satisfaction and streamlined workflow in various healthcare settings.Reported facilitators to tool usage included relevance to practice, accessibility, and facilitating multidisciplinary action, making these tools practical and time-efficient for healthcare settings.Barriers reported included being time-consuming, irrelevant to practice, difficult to use, and lack of organizational engagement, highlighting areas for improvement.The co-design approaches varied across studies, reflecting different levels of stakeholder engagement and adoption of co-design methodologies.

Strengths and limitations
This study adhered to the PRISMA statement and followed the registered study protocol in PROSPERO.This was the first systematic review to summarize how website-based tools can support health professionals with quality improvement.
However, there are several limitations.Inclusion of qualitative studies prevented the ability to perform data analyses and overall quality for syntheses was low.Heterogeneous study designs, outcomes, and methodologies limited generalizability of findings.Some studies lacked the details regarding methodology, including justification for chosen designs.Narrative reviews may have limitations in terms of objectivity.

Interpretation within the context of the wider literature
Many tools in the current review helped the user to identify gaps in practice and this was suggested to be useful in QI, with 'relevance to practice' identified as a facilitator to tool usage.Identifying gaps in practice is a common feature across many QI projects in professional settings.For example, an auditing tool for surgical QI was designed to assess compliance with infection-related process measures and identify gaps in measure implementation.It was found that the tool was useful in identifying gaps and quarterly compliance improved for 80% of process measures [34].Another tool for assessing resident's competence was developed to provide data on gaps in knowledge, which could be used to guide curriculum development [35].Identifying the gaps, especially between evidence and practice or policy-making, is one of the crucial first steps in knowledge translation [36].This is likely why tools that can identify gaps in practice prove to be valuable in QI initiatives and can be used during the formative planning process.
The findings of the current review show that many websitebased tools have been developed to digitalize existing QI processes, and 'accessibility' was identified as a facilitator to tool usage.A study examining digital tools for patient monitoring in oncology care found that digitalization allowed for nurse practitioners to efficiently extend and improve symptom management [37].Another study found that a digital tool for a home-based blood pressure monitoring programme was effective and timely in hypertension management [38].In recent years, digital transformation has been observed across many professional sectors and this has been associated with numerous advantages.For instance, the implementation of the Electronic Health Record has led to improvements in quality of care, reduction in medical errors, and increased adherence to clinical guidelines [39].Considering how important QI is in the workforce, it was not surprising that many tools analysed in the current review sought to digitalize existing processes.
Although digitalization has many advantages, there are limitations that should be considered.In the current review, a barrier to tool usage was the tool being time consuming or difficult to use.Similarly, a systematic review examining web-based interventions for weight loss highlighted that one of the challenges was engagement and retention.The authors suggested that there is a need to investigate components of web-based tools that can maintain users' motivation and interest [40].A study exploring digital tools to deliver physical activity advice identified that the biggest influence was having the skills to utilize the tool [41].These observations, including the findings of the current review, suggest that digital literacy is an important consideration when developing digital tools or interventions and addressing these barriers may assist in maximizing user engagement and successful tool implementation.
Another key QI process that the tools in the current review adopted was contributions to professional development.Education/training resources were also identified as one the common tool features.Many QI tools are designed to assist with professional development.For example, a digital application designed for teachers to improve their students' linguistic competence assisted the teacher to self-assess their classes, make decisions, and facilitate practice changes through professional development.The use of the tool led to improved competencies for both the teachers and students [42].A study that reported the development of a self-assessment tool for dental faculty to map professional growth identified the lack of defined faculty competencies in medical and dental education.Subsequently, a tool was developed and could be used as part of professional development [43].Literature across all professions have demonstrated great benefits in undertaking professional development and this highlights why it is an ongoing requirement of practice for some.For example, continuing professional development is a requirement for nurses and has been directly linked to nurses' career satisfaction and continuous growth in their practice [44].
In the current review, almost all tools were co-developed with stakeholders.This finding is further supported by 'facilitating multidisciplinary action' being identified as a facilitator of tool usage.Similarly, a QI project to improve ambulatory care for patients with musculoskeletal disorders focused on collaboration between general practitioners and specialists [45].The study found that collaborative care was associated with a lower risk of osteoarthritisrelated hospitalization, higher participation in exercise interventions, and more frequently prescribed physical therapy.The benefits of collaboration can be attributed to many factors, such as the opportunity to learn from others, access to new resources, increased productivity, and shared goals [46].
The current review identified lack of organizational engagement as a barrier, which included reluctance to change.A study exploring resistance towards changes among healthcare staff identified several reasons for reluctance, including personal reluctance, misunderstanding of project/initiative aims, and a dislike of the methods by which projects have been promoted [47].A QI initiative for educational programmes found that there was reluctance among some teachers due to their uncertainty about potential benefits that could arise from the initiative [48].In the current review, the qualitative data indicated factors, such as resource constraints, low staffing, and managerial support as barriers.These findings indicate that the reasons for lack of organizational engagement are multifactorial, and further research exploring this barrier would be beneficial.

Implications for policy, practice, and research
The findings offer valuable insights for future development of QI website-based tools.It is expected that more digital tools will be developed to drive practice improvements and the findings of this review can be useful in guiding the development process.Recommendations include ensuring tools are co-developed with healthcare professionals, focusing on practical usability, and addressing common barriers to enhance engagement and effectiveness in improving healthcare quality.The insights into the barriers and facilitators of tool usage can be broadly applied to any project that seeks to improve practice.Randomized controlled trials are warranted to provide objective evidence of tool efficacy.

Conclusion
The findings emphasize the importance of co-development with healthcare professionals, practical usability, and addressing barriers to enhance engagement and effectiveness of QI digital tools.There is a lack of randomized controlled trials on the efficacy of these tools.Future work should address this knowledge gap.

Figure 1
Figure 1 Study flow diagram of search results and the selection process.

Table 1 .
Characteristics of included studies.

Table 2 .
Summary of tool outcome measures and assessment methods.

Table 3 .
Website-based tools grouped by primary QI process/mechanism.